Skip to content

feat: execute droptable and reopen tasks in batch#421

Merged
thweetkomputer merged 3 commits intomainfrom
batch-global-request-zc
Mar 18, 2026
Merged

feat: execute droptable and reopen tasks in batch#421
thweetkomputer merged 3 commits intomainfrom
batch-global-request-zc

Conversation

@thweetkomputer
Copy link
Collaborator

@thweetkomputer thweetkomputer commented Mar 17, 2026

Here are some reminders before you submit the pull request

  • Add tests for the change
  • Document changes
  • Reference the link of issue using fixes eloqdb/eloqstore#issue_id
  • Reference the link of RFC if exists
  • Pass ctest --test-dir build/tests/

Summary by CodeRabbit

  • New Features

    • Added a new configuration option, max_global_request_batch (default 1000), to limit concurrent per-partition requests for global operations.
  • Refactor

    • Improved scheduling and concurrency for table truncation, archive, and reopen workflows with bounded in-flight requests, coordinated completion, and aggregated error reporting.

@coderabbitai
Copy link

coderabbitai bot commented Mar 17, 2026

Walkthrough

Adds a new KvOptions field max_global_request_batch and refactors DropTable, GlobalArchiveRequest, and GlobalReopenRequest to schedule per-partition subrequests via shared schedule-state machines that cap concurrent in-flight requests and aggregate per-subrequest results.

Changes

Cohort / File(s) Summary
Configuration Structure
include/kv_options.h, src/kv_options.cpp
Add uint32_t max_global_request_batch = 1000 to KvOptions; load from INI and include in equality comparison.
Global Operation Scheduling
src/eloq_store.cpp
Refactor DropTable, GlobalArchiveRequest, and GlobalReopenRequest to use schedule-state classes (per-path) that manage per-partition subrequest enqueueing, bounded in-flight concurrency using max_global_request_batch, per-subrequest result aggregation, and coordinated completion signaling.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant EloqStore
  participant Scheduler as ScheduleState
  participant Worker as ExecAsync/PartitionHandler
  Client->>EloqStore: Send global operation (archive/reopen/drop)
  EloqStore->>Scheduler: Create ScheduleState (targets[], max_inflight)
  Scheduler->>Worker: Launch up to max_inflight subrequests
  Worker-->>Scheduler: Subrequest result (ok / error)
  Scheduler-->>Worker: Launch next subrequest (until all started)
  Scheduler->>EloqStore: All subrequests completed (aggregate result)
  EloqStore-->>Client: Respond with final status
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • eatbreads

Poem

🐰 I hop through batches, steady and bright,
Scheduling requests by soft moonlight.
Partitions pace, no frantic race,
Errors gathered, then set in place.
A rabbit's batch—calm, measured delight.

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description is just a template checklist with no substantive content—it contains only unchecked boxes for tests, documentation, issue references, RFC links, and test execution, providing zero meaningful information about the changes. Replace the template checklist with actual implementation details: describe what was changed, why batching was added, how max_global_request_batch works, and the impact on droptable/reopen operations.
Docstring Coverage ⚠️ Warning Docstring coverage is 7.69% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: introducing batch execution for droptable and reopen tasks, which aligns with the changeset's core objective of implementing batched global request handling.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch batch-global-request-zc
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/eloq_store.cpp (1)

933-938: Make the global scheduler bounded end-to-end.

Each handler now limits concurrent dispatch, but it still allocates one subrequest object per partition before the first send. That keeps heap growth proportional to partition count and repeats the same scheduler skeleton three times. A small shared helper that keeps only a bounded pool of live subrequests and creates them lazily would remove the duplication and make max_global_request_batch cover memory as well as concurrency.

Also applies to: 940-1008, 1176-1187, 1192-1260, 1398-1407, 1409-1496

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/eloq_store.cpp` around lines 933 - 938, The current handlers eagerly
allocate one subrequest per partition (e.g., creating TruncateRequest instances
and pushing into req->truncate_reqs_) which allows heap growth proportional to
partition count; refactor by introducing a small shared helper (e.g., a
BoundedSubrequestPool or make_lazy_subrequest_generator) that maintains at most
max_global_request_batch live subrequests, creates TruncateRequest (and the
other subrequest types used in the other handlers) lazily when about to
dispatch, and recycles/completes them when a dispatched subrequest finishes;
replace the per-partition eager loops that push into truncate_reqs_ with
iteration that obtains subrequests from this pool and sends them, and reuse the
same helper across the three similar handler sites (the ranges called out) so
memory is bounded by max_global_request_batch as well as concurrency.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/kv_options.cpp`:
- Around line 178-182: When loading max_global_request_batch in KvOptions (the
reader.HasValue/GetUnsigned block for sec_run and max_global_request_batch),
ensure a zero value is not accepted: after reading the unsigned value, normalize
0 to 1 (or reject by returning an error); additionally add a defensive check in
ValidateOptions() to assert max_global_request_batch > 0 and fail validation if
not. Update the reader.GetUnsigned usage for max_global_request_batch and the
ValidateOptions() logic in KvOptions to reference the same semantic (must be >
0).

---

Nitpick comments:
In `@src/eloq_store.cpp`:
- Around line 933-938: The current handlers eagerly allocate one subrequest per
partition (e.g., creating TruncateRequest instances and pushing into
req->truncate_reqs_) which allows heap growth proportional to partition count;
refactor by introducing a small shared helper (e.g., a BoundedSubrequestPool or
make_lazy_subrequest_generator) that maintains at most max_global_request_batch
live subrequests, creates TruncateRequest (and the other subrequest types used
in the other handlers) lazily when about to dispatch, and recycles/completes
them when a dispatched subrequest finishes; replace the per-partition eager
loops that push into truncate_reqs_ with iteration that obtains subrequests from
this pool and sends them, and reuse the same helper across the three similar
handler sites (the ranges called out) so memory is bounded by
max_global_request_batch as well as concurrency.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 63e013d1-96b6-4731-94f5-cf2bba06162e

📥 Commits

Reviewing files that changed from the base of the PR and between 24483fb and c85978c.

📒 Files selected for processing (3)
  • include/kv_options.h
  • src/eloq_store.cpp
  • src/kv_options.cpp

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/eloq_store.cpp (1)

945-1013: LGTM - Well-structured batch scheduling pattern.

The DropTableScheduleState implementation is sound:

  • Proper use of shared_from_this() for safe callback lifetime management
  • Correct memory ordering: acq_rel on pending_ ensures visibility of first_error_ writes
  • Graceful failure handling with callback clearing before SetDone

Note that DropTableScheduleState, ArchiveScheduleState, and ReopenScheduleState are nearly identical. Consider consolidating into a template or helper to reduce duplication in a follow-up PR.
,

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/eloq_store.cpp` around lines 945 - 1013, The three nearly identical
scheduler structs (DropTableScheduleState, ArchiveScheduleState,
ReopenScheduleState) should be consolidated into a single templated or generic
helper to remove duplication: extract the common members (store, req, total,
next_index) and methods (HandleTruncateResult, OnTruncateDone, ScheduleNext)
into a template class or a policy-based helper that accepts the specific request
type (e.g., TruncateRequest container/accessor) and callback behavior; update
the call sites to instantiate the template with the appropriate request type and
preserve current semantics (shared_from_this usage, memory orderings, error
handling) so functionality and lifetime management remain unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/eloq_store.cpp`:
- Around line 945-1013: The three nearly identical scheduler structs
(DropTableScheduleState, ArchiveScheduleState, ReopenScheduleState) should be
consolidated into a single templated or generic helper to remove duplication:
extract the common members (store, req, total, next_index) and methods
(HandleTruncateResult, OnTruncateDone, ScheduleNext) into a template class or a
policy-based helper that accepts the specific request type (e.g.,
TruncateRequest container/accessor) and callback behavior; update the call sites
to instantiate the template with the appropriate request type and preserve
current semantics (shared_from_this usage, memory orderings, error handling) so
functionality and lifetime management remain unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 38f1422d-1e46-4451-856b-6e66f4d6e42a

📥 Commits

Reviewing files that changed from the base of the PR and between c85978c and c7d1941.

📒 Files selected for processing (1)
  • src/eloq_store.cpp

@thweetkomputer thweetkomputer merged commit f4a4aff into main Mar 18, 2026
6 checks passed
@thweetkomputer thweetkomputer deleted the batch-global-request-zc branch March 18, 2026 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants